Videos distributed via TV broadcast and Videos stored on recording media, such as video tapes, DVDs, or the like, are often interlaced videos in which even field images and odd field images decimated every other line in the vertical direction from a progress video are interleaved. The relation between original progressive image F(t) and interlaced image Fi(t) at the time t is expressed by the following equation (1).
                    [                  Equation          ⁢                                          ⁢          1                ]                                                                      Fi          ⁡                      (                          x              ,              y              ,              t                        )                          =                  {                                                                      F                  ⁡                                      (                                          x                      ,                      y                      ,                      t                                        )                                                                                                if                  ⁡                                      (                                                                  y                        ⁢                                                                                                  ⁢                        mod                        ⁢                                                                                                  ⁢                        2                                            =                                              t                        ⁢                                                                                                  ⁢                        mod                        ⁢                                                                                                  ⁢                        2                                                              )                                                                                                      Null                                            else                                                                        (        1        )            
where F(x,y,t) and Fi(x,y,t) represent pixel values F(t) and Fi(t) on the coordinate (x, y), respectively. y mod x is an arithmetic symbol representing the remainder of y/x.
However, video display units, such as LCDs or plasma displays, display progressive videos. In order to display interlaced videos on such video display units, progressive images must be generated by restoring Null pixels on decimated lines of interlaced images, expressed by the equation (1), via interpolation. The interpolation process is generally called as the interlacing to progressive conversion (IP conversion, De-interlacing). Hereinafter, Null pixels restored via interpolation are called as interpolated pixels.
As one of IP converting methods, there is the motion compensation IP conversion. One of motion compensation IP conversion methods can provide high performance interpolation execution. That is, when the progressive image Fp(t) at the time t is restored from an interlaced image, a first progressive image Fp1(t) calculated based on pixel values around respective interpolated pixels at the time t within an interlaced image Fi(t) is synthesized with a second progressive image Fp2(t), obtained by utilizing motion vectors by which respective interpolated pixels are associated with the corresponding pixels within interlaced image at other time and calculating based on pixel values of the corresponding pixels within an interlaced image at other time. Hereinafter, the method of calculating a first progressive image from an interlaced image at the same time is called as a first progressive image calculating method. The method of using motion vectors and calculating a second progressive image based on image pixels at other time is called as a second progressive image calculating method.
As the first progressive image calculating method, there are a simple linear interpolation method, expressed by the equation (2), and an edge adaptive interpolation method, expressed by the equation (3), (non-patent document 1 and patent document 1). Fp1(x,y,t) is the pixel value of an interpolated pixel at the coordinate (x,y) in the first progressive image Fp1(t) at the time t. m in the equation (3) is p minimizing the equation (4) within a predetermined range −Φ≦p≦Φ. Hereinafter, the range −Φ≦p≦Φ is called as a search range Φ. Moreover, referring to the non-patent document 2, the changes in luminance of the upper and lower lines within the search range by the edge adaptive interpolation method in respective interpolated pixels based on the luminance conversion of peripheral pixels are classified into five patterns, as shown in FIG. 8. That is, five patterns includes (1) one characteristic being flat, (2) both characteristics monotonously increasing and decreasing in the same direction, (3) both characteristics being convex in the same direction, (4) one characteristic decreasing and increasing and the other characteristic being convex, (5) others. The search range Φ changes adaptively to the maximum range such that the pattern of a change in luminance of the upper or lower line within the search range belongs to any one of the patterns (1) to (4). Hereinafter, patterns in luminance change upper and lower lines within a classified search range are called as luminance change patterns of upper and lower lines.
                    [                  Equation          ⁢                                          ⁢          2                ]                                                                      Fp          ⁢                                          ⁢          1          ⁢                      (                          x              ,              y              ,              t                        )                          =                                            Fi              ⁡                              (                                  x                  ,                                      y                    -                    1                                    ,                  t                                )                                      +                          Fi              ⁡                              (                                  x                  ,                                      y                    +                    1                                    ,                  t                                )                                              2                                    (        2        )                                [                  Equation          ⁢                                          ⁢          3                ]                                                                      Fp          ⁢                                          ⁢          1          ⁢                      (                          x              ,              y              ,              t                        )                          =                                            Fi              ⁡                              (                                                      x                    -                    m                                    ,                                      y                    -                    1                                    ,                  t                                )                                      +                          Fi              ⁡                              (                                                      x                    +                    m                                    ,                                      y                    +                    1                                    ,                  t                                )                                              2                                    (        3        )                                [                  Equation          ⁢                                          ⁢          4                ]                                                                      Sub          ⁡                      (            p            )                          =                                                      Fi              ⁡                              (                                                      x                    -                    p                                    ,                                      y                    -                    1                                    ,                  t                                )                                      -                          Fi              ⁡                              (                                                      x                    +                    p                                    ,                                      y                    +                    1                                    ,                  t                                )                                                                                  (        4        )            
The second progressive image calculation method utilizes motion vectors establishing correspondences between respective interpolated pixels and related pixels within an interlaced image at other time and calculates a second progressive image based on the pixel values of the corresponding pixels within an interlaced image at other time.
As to motion vector calculation, there are a block matching method and a gradient method. When a motion vector in the interpolated pixel on the coordinate (x,y) at the time t is (dx,dy), the pixel value Fp2(x,y,t) of the second progressive image in the interpolated pixel is calculated by the equation (5). Moreover, there is the method of detecting the horizontal motion dx′ and calculating the pixel value of the interpolated pixel (x,y), according to the equation (6).
                    [                  Equation          ⁢                                          ⁢          5                ]                                                                      Fp          ⁢                                          ⁢          2          ⁢                      (                          x              ,              y              ,              t                        )                          =                                            Fi              ⁡                              (                                                      x                    -                    dx                                    ,                                      y                    -                    dy                                    ,                                      t                    -                    1                                                  )                                      +                          Fi              ⁡                              (                                                      x                    +                    dx                                    ,                                      y                    +                    dy                                    ,                                      t                    +                    1                                                  )                                              2                                    (        5        )                                [                  Equation          ⁢                                          ⁢          6                ]                                                                      Fp          ⁢                                          ⁢          2          ⁢                      (                          x              ,              y              ,              t                        )                          =                                            Fi              ⁡                              (                                                      x                    -                                          dx                      ′                                                        ,                  y                  ,                                      t                    -                    1                                                  )                                      +                          Fi              ⁡                              (                                                      x                    +                                          dx                      ′                                                        ,                  y                  ,                                      t                    +                    1                                                  )                                              2                                    (        6        )            
When a motion vector can be obtained accurately, the second progressive image to be calculated using the motion vector is generated with high interpolation accuracy.
However, in the interlacing to progressive conversion, it is difficult generally to obtain the motion vectors for all interpolated pixels accurately. For example, when a background object, shielded by a foreground object, appears due to a movement of the foreground object, the movement of the background object at appearance time cannot be obtained accurately because the location of the background object before appearance cannot be known. Therefore, the reliability of motion vector estimation is calculated. The first progressive image and the second progressive image are synthesized based on the motion estimation reliability. In this case, if the motion vector estimation reliability is high, the synthesis ratio of the second progressive image is set to be high and if the motion vector estimation reliability is low, the synthesis ratio of the second progressive image is set to be low. Hereinafter, the reliability of motion vector estimation is called as motion estimation reliability.
Referring to, for example, the patent document 2, the reliability of motion vector estimation is calculated based on the sum of difference absolute values (SAD, Sum of Absolute Differences). The sum is one factor representing the similarity between an interested interpolated pixel and the corresponding pixel, used when a motion vector is obtained through the block matching method. In the patent document 2, the motion vector estimation reliability is higher when the sum of difference absolute values is small while is smaller when the sum of difference absolute values is large.
An image processing apparatus for performing motion compensation IP conversion related to the present invention will be explained by referring to FIG. 10. FIG. 10 is a block diagram illustrating the configuration of an image processing apparatus that performs motion compensation IP conversion, according to the present invention. Referring to FIG. 10, the image processing apparatus 1 includes a first progressive image calculation means 11, a second progressive image calculation means 12 and a progressive image synthesis means 13. The image processing apparatus receives the interlaced images Fi(t−1), Fi(t) and Fi(t+1) at the time t−1, t, and t+1, respectively, and outputs the progressive image Fp(t) at the time t.
The first progressive image calculation means 11 calculates the first progressive image using the first progressive image calculation method.
The second progressive image calculation means 12 calculates the second progressive image using the second progressive image calculation method and outputs the sum of the second progressive image and the sum of difference absolute values.
The progressive image synthesis means 13 synthesizes the first progressive image and the second progressive image based on the motion estimation reliability calculated based on the sum of difference absolute values.    Patent document 1: Japanese patent Laid-open publication No. Hei4-355581    Patent document 2: Japanese patent Laid-open publication No. 2006-174123    Non-patent document 1: “Deinterlacing—an overview”, De Haan, G, Betters, E. B, Proceedings of the IEEE, Volume 86, Issue 9, September 1998 page(s): 1839-1857    Non-patent document 2: “Edge adaptive interlace to progressive conversion method based on a brightness change about peripheral area”, Toda, Tukada, and Inoue, Sixth Information Science Technology Forum, 1-034, 2007